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This paper is a continuation of the preceding article which introduced the reader to the general concepts 
of ruggedness testing. The current paper describes the effects of interactions on the measurement process, and 
presents procedures for the separation of the main effects from the two-factor interactions. The general 
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calculations. A number of examples of glass electrode measurements of pH of dilute acid solutions are used 
to illustrate ruggedness testing procedures. 
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Introduction 

This paper is a continuation of the preceding (Part I) 
article which introduced the general principles of rug- 
gedness testing. To be read in conjunction with Part I, it 
describes the effects of interactions on a measurement 
process and presents procedures for separating main ef- 
fects and two-factor interactions. 

Interactions and That Confounded Confounding 

From Part I we know that an N measurement experi- 
ment can be used to determine N-l main factors, pro- 
vided the interactions are small. It is usually the case, in 
experiments involving well-behaved functions of the 
measurement variables, that when the main effects are 
small the associated interactions are very small. The 
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interactions are, in effect, the non-ideal departures from 
a simple additive model consisting of only constant main 
effects. Nevertheless, situations occasionally arise in 
which interactions are important. 

In an eight-run, seven-factor experiment each main 
effect is confounded with 15 different possible inter- 
actions. Of the 15 interactions, the number and types are 
as follows: 3 two-factor, 4 three-factor, 4 four-factor, 
3 five-factor, and 1 six-factor. Table 5, 1 which corre- 
sponds to theYates-Youden design (see table 2 of part I), 
shows each of the main effects and the associated two- 
and three-factor interactions. 

Table 5. 1 Interactions associated with the main effects. 



A 


B 


C 


D 


E 


F 


G 


-BD 


-AD 


-AE 


-AB 


-AC 


-AG 


-AF 


-CE 


-CF 


-BF 


-CG 


-BG 


-BC 


-BE 


-FG 


-EG 


-DG 


-EF 


-DF 


-DE 


-CD 


BCG 


ACG 


ABG 


ACF 


ABF 


ABE 


ABC 


BEF 


AEF 


ADF 


AEG 


ADG 


ACD 


ADE 


CDF 


CDE 


BDE 


BCE 


BCD 


BDG 


BDF 


DEG 


DFG 


EFG 


BFG 


CFG 


CEG 


CEF 



1 This paper (Part II), being a continuation of Part I, extends Part Fs 
numbering system for tables and equations; its citations are listed in 
Part Fs References section. 



The Yates-Youden design (and the Plackett-Burman 
designs of a size such that N =2\ where k is a positive 
integer) allow a relatively easy separation and determi- 
nation of the more important confounding interactions. 
These designs allow one to use the Multiplication Rule 
for signs. The Multiplication Rule [4] 1 states that the 
pairwise multiplication of like signs produces a (+) and 
that of unlike signs produces a (-). Thus, looking at 
table 2 of Part I, the row pairwise multiplication of the 
signs for columns B and D produces the following 
column for the BD interaction: 



BD 

+ 
+ 

+ 
+ 



(see table 3 of Part I), and a third set of pH mea- 
surements which was made with all levels of the design 
reversed. Let us now consider the combined results 
from the first and third sets of measurements. 

An examination of the signs of table 6, and the use of 
the Multiplication Rule, will show that the two-factor 
interactions —BD, —CE, and — FG (which were 
grouped together in column 1 of table 5) still have an 
identical sign pattern but that this pattern is now 
different from the A main effect. The — BD inter- 
action has the following sign pattern 

( h + + H h + + +). One can see 

that the last half of the interaction sign pattern is a 
repetition of the first half whenever an even-number of 
factors is multiplied together, but that the last half has a 
sign reversal whenever an odd-number of factors is mul- 
tiplied together. 

Table 6. Designs and test results. 



Factor 
A B C D E F G 



Observed pH 
{milli-pH units) 



Note that this column is the exact opposite of the signs 
of column A , given in table 2. Thus, — BD is the same as 
A . It is inseparable from A in the eight-run, seven-factor 
experiment since the values of the eight measurements 
are combined in an identical manner. Similar multi- 
plications of signs shows that A=— CE=— FG. Multi- 
plication of signs of the rows of columns "BC" and G 
produces the three-factor interaction BCG which is ob- 
served to be the same as factor A . Column "BC" can be 
simply obtained by using minus column F (see table 5). 
The confounding of all higher order interactions can be 
obtained by an extension of this general procedure. 

If we wish to protect ourselves from misin- 
terpretations due to large interactions, we must make 
more than N measurements for determining the N-l 
main factors. To evaluate the main effects and all inter- 
actions, we must do the full factorial experiment. For 
seven factors this requires 128 measurements. Usually, 
however, one does not have to go this far. A reasonable 
compromise experiment consists of making two sets of 
N measurements which allow the separation of each of 
the main effects from the two-factor interactions. This 
compromise, however, does not separate among each of 
the two-factor interactions, and in addition it assumes 
that three-factor and other higher order interactions are 
unimportant. If we demand more information, then we 
have no choice but to make more measurements! 

Let us again consider a seven factor pH experiment 
involving 2N (=16) measurements. This time we will 
use the previously reported first set of pH measurements 



--+- + + + 

- + - + - + + 

- + + + + -- 
+ -- + + - + 
+ -+ + -+- 
+ + -- + +- 
+ + +--- + 

+ + + + + + + 
+ + - + --- 
+ -+- + -- 
+ ----+ + 

- + +--+- 

- + -- + - + 
--+ + -- + 
--- + + +- 



2904 
3015 

3006 
2964 
2999 
3055 
3049 
2949 



3040 
2931 
2978 
3030 
2967 
2911 
2874 
2979 



From table 6, and the Multiplication Rule, one can see 
that the three two-factor interactions within each col- 
umn of table 5 are not separable from one another, but 
that they are separable from the main effects. The three- 
factor interactions are not separable from the main ef- 
fects. A further consideration of table 6 will show that 
the main effects and their odd-numbered interactions are 
not separable from one another, but that they are sepa- 
rable from all of the even-numbered interactions. The 
nearest higher order interaction contamination for ei- 
ther the odd- or the even-numbered interactions is now 
two-factor multiples distant. If the magnitude of the 
interactions decreases as one goes toward higher order 
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interactions, then one has achieved a practical sepa- 
ration (isolation) of the main effects and of the groups of 
two-factor interactions. 

Main effects A-G can be calculated from the data of 
table 6 by use of eq (1) (from Part I). The calculated 
respective effects are +51, —2, +4, +6, + 27, +79, and 
—0.4 milli-pH units. The two-factor interactions are cal- 
culated in the same manner as the main effects. Note that 
the value of the "new N" in eq (1) is the combined N 
from both sets of measurements (new N= 16). As shown 
above, the sign pattern for the — BD interaction is 

( h + + + h + + +). The value for 

the combined (BD, CE, FG) interactions is +11 milli- 
pH units. The other two-factor interactions can be cal- 
culated in a similar manner. Finally, we note from table 
6 that if an offset had occurred between the first and 
third set of measurements, it would not affect the calcu- 
lations of the main effects or the interactions. This im- 
munity to offsets between the different sets of mea- 
surements is a consequence of using the 
Plackett-Burman based design. The PB-design will al- 
ways have an equal number of positive and negative 
signs within each set so that the absolute level of the sets 
of measurements will not affect the calculations. 

Short-Cut Calculations 

All of the ruggedness testing calculations are concep- 
tually quite simple, but are tedious to perform. Hand 
calculators that have at least nine memory registers al- 
low short-cuts that minimize the arithmetic operations 
and the keying of the data. We will assume here that our 
calculations are made on sets of eight measurements. Let 
the average of these measurements be X Starting from 
eq (1) of Part I, the derivation of the short-cut method 
is as follows: 



Effect A = 



2A(+) IA(-) 



N/2 



N/2 



2A 

N/2 



(+), pM(+) Mi±) 

V2 ~*~[ N/2 N/2 



2A(-) 



N/2 



ForN = 8, 



Effects = 42 l T (+) -2Z 
N 



Effect^ =2A(+)/2-2X 



(9) 



(9a) 



Let us now rewrite the table 2 design of Part I for the 
first set of eight pH measurements, substituting the or- 
dered measurement numbers for the positive signs. 



For the set of eight measurements, one keys the mea- 
surements into memory registers 1-8, respectively, and 
then calculates the last term of eq (9a) which is two 
times the average of the eight measurements. This quan- 
tity is stored in memory register 9, In order to minimize 
the chance of error, it is advisable to use the mea- 
surement results that are stored in memory registers 1-8 
to calculate this latter quantity. One then simply uses eq 
(9a) and the columns of table 7 to calculate the various 
effects: 

Effect A = (Registers 5 + 6 + 7 + 8)/2 - Register 9 

= +40.75 milli-pH 
Effect B = (Registers 3 + 4 + 7 + 8)/2 - Register 9 

= -1.25 milli-pH. 

Table 7. Relabeled design and test results. 



Factor 
A B C D E F G 



Observed pH 
(milli-pH units) 



- - 2 - 2 2 2 

_ 3 - 3 - 3 3 

_ 4 4 4 4 - - 

5 - - 5 5 - 5 
6-66-6- 

7 7 _ _ 7 7 _ 

8 8 8 - - - 8 



2904 
3015 
3006 
2964 
2999 
3055 
3049 
2949 

Average 2992.625 



The reverse sign PB-design listed in the bottom of table 
6 can be similarly rewritten and used with eq (9a) to 
again calculate effects A-G. 

Table 8 lists the calculated effects from the three sets 
of eight pH measurements that have been previously 
reported in tables 3 and 6. The actual, chronological 
order used for making our measurement sets consisted 
of the table 2 design of Part I, the reverse-sign design, 
the repeat table 2 design, and occasionally a repeat of 
the reverse-sign design. The labeling from our pre- 





Table 8. 


Calculated effects. 














(milli-pH uni 


ts) 






Factor 


A 


B 


C 


D 


E 


F 


G 


Set 1 


41 


-1 


6 


27 


28 


11 


-1 


Set 2 


62 


-3 


2 


-16 


26 


80 





Set 3 


48 


-7 


11 


14 


23 


85 


3 



viously referenced data sets 2 and 3, will henceforth be 
reversed to conform to the chronological order. Thus, 
set 2 will now refer to the reverse-sign design, and set 3 
to the repeat table 2 design. 

We see that the set 2, reverse-sign PB-design, gives 
slightly different results. Note that the use of the Multi- 
plication Rule on the reverse-sign PI? design (listed in 
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the bottom half of table 6), results in positive (rather 
than negative) two-factor interactions which are con- 
founded with the main effects. For example, 
A = +BD = +CE = +FG. Let us take averages for the 
table 8 results for sets 1 and 2. For each of the averages 
of sets 1 and 2, the two-factor (and other even-number 
factor interactions) drop out. By similar reasoning the 
differences between sets 1 and 2, when divided by two, 
yields the separated even-number factor inter- 
actions.The set 1 and 2 averages and average differences 
are listed in table 9. 



ages and average differences (as shown in table 10). 
These latter averages and average differences are the 
separated, and relatively uncontaminated, main effects 
and interactions. 

For PB-designs with N = 8, the standard deviation of 
a single measurement is obtained by taking the square- 
root of the average of the above calculated squared 
differences of the "contaminated" effects (see eq (7) of 
Part I). For the current experiment s equals V384/7 or 
7.4 milli-pH units. It has 7 degrees of freedom associated 
with it. 



Table 9. Averages and average differences for effects. 




Factor A 


For Data Sets 1 and 2 

(milli-pH units) 

B C D E F 


G 


Average 52 
Avg. Difference 1 1 


-2 4 6 27 79 
_1 _2 -22 -1 2 


-I 
1 



The table 9 results for sets 1 and 2 are the same as the 
results obtained by the more tedious, direct calculations. 
The calculated effects from sets 2 and 3 could also be 
used to produce results comparable to table 9. It should 
be obvious that better (more stable) estimates can be 
obtained by first pre-averaging the effects from sets 1 
and 3 before making the combined calculation with the 
set 2 data to produce the averages and average differ- 
ences. The results from such calculations are given in 
table 10. For simplicity of presentation, these averages 
and average differences will hereafter be called the 
Main Effects and the Two-Factor Interactions, re- 
spectively. 

Let us now summarize the short-cut calculations: for 
each data set use eq (9) or (9a), and its associated design 
table (such as table 7) to calculate the "contaminated" 
effects (as shown in table 8). Where possible, for like- 
sign designs, calculate the pre-averages. Also, for the 
like-sign designs calculate the squared differences of the 
"contaminated" effects (their use will be described in 
the next paragraph). The pre-averaged effects from the 
like- and reverse-sign designs are used to calculate aver- 



Judging the Main Effects and Interactions 

To help decide if the main effects and two-factor 
interactions are real, or if they may simply be due to 
imprecisions in the measurements, let us once again use 
the t -statistic. 

effect under test 



/=- 



^effect under test 



To determine the /-value we must evaluate the denomi- 
nator of the equation. Since the main effects (ME) and 
the two-factor interactions (2FI) are calculated by ta- 
king either the averages or average differences from the 
same sets of data, the standard deviation of the ME and 
the 2FI will be the same. For the current example, the 
ME and the 2FI are calculated as follows: 



n 



Sets 1 + 3 



± Set 2 



The recognition of the form of the above calculations, 
and the use of the square of eq (2b), allow the evaluation 
of the standard deviation of the ME or the 2FI. 



s(MEor2FI)=2 



4s 2 /8+4s 2 /8 , 4y 2 /8 

i2 + 1 



=V3s/4 (10) 



/ = 



4 X (effect under test) 



V3 



s. 



Table 10. Main effects and two-factor interaction. 



Factor 



For Data Sets [Avg. (1 and 3)] and 2 
(milli-pH units) 
A B C D E F 



Legend for + Factors 



Main Effects 


53 


-4 


5 


2 


26 


81 


1 


Two-Factor Interactions 


9 


1 


-3 


-18 





-1 


-1 



A = 


25 °C 


B = 


solution stirred 


C = 


0.5 mL dilution 


D = 


1 cm electrode immersion 


E = 


NaN0 3 added 


F = 


KC1 added 


G = 


pH measured at 10 min 
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For the current example s equals V384/7 or 7.4 milli-pH 
units. The observed f -value is as follows: 

/ 7 =0.31 X (effect under test) . 

For the main effect A ( = 53) we have: 

; 7 =0.31X53 = 16. 

We see from table 10 that the main effects A, E, and 
F, and the two-factor interactions under columns^ and 
D are statistically significant. This is very similar to our 
prior conclusions in Part I, except that we now dis- 
tinguish between the main effects and the two-factor 
interactions. 

Results From the Other Dilute Acid Solutions 

A total of six different dilute acid solutions covering 
a pH range from 3.0 to 5.0 was tested. The purpose of 
these tests with the six solutions was to further evaluate 
the measurement procedures and to determine a prac- 
tical upper limit for the pH measurements. Toward the 
end of the tests, the original glass electrode was broken 
and a second glass electrode of the same model and 
manufacturer was used as a replacement. This acci- 
dental breakage gave additional practical insight into 
ruggedness testing. 

Some information regarding the six acid solutions is 
given in table 11. 

The main design (table 2 of Part I) and the reverse- 
sign design were run with all six solutions. A standard 
pH=4.00 buffer was used to calibrate the pH meter 
before making each set of eight measurements. In many 
cases the designs were repeated and the standard devi- 
ations for a single measurement, s t were calculated. In 
accordance with the chronological order of the mea- 
surements, the main designs (table 2 of Part I) are la- 
beled as sets 1 and 3, and the reverse-sign designs are 
labeled as sets 2 and 4. The calculated pre-averages for 
the like-sign designs are listed in table 12. 

While making the measurements on sets 1, 2, and 3 of 
HC1 solution 6, it was clear that there was great instabil- 
ity in the measurements. This instability is reflected in the 



table-displayed standard deviation of 107 milli-pH units 
for solution 6. The listed pre-averages for solution 6 are 
also wild. The measuring equipment was operating 
properly. The problem appears to be associated with 
solution 6. Let us temporarily set the solution 6 results 
aside and examine the rest of table 12. 

For glass electrode #1 there are rather large effects 
for the addition of either NaN0 3 or KC1 
(Factors E and F). The pre-averages appear to be 
larger for the H 2 S0 4 solutions (1,2) than for the HC1 
solutions (3-5). For glass electrode #2, the pre-aver- 
ages for NaNQ3 and /or KC1 appear to be much smaller 
and are the same magnitude for both the H2SO4 and the 
HC1 solutions. 

The main effects and two-factor interactions were 
next calculated from the pre-averages by the short-cut 
procedures (see table 13). Average values for the stan- 
dard deviation of a single measurement are also shown. 
The general calculations described by eq (10), and the 
table-displayed s values, provide a method by which we 
can judge the main effects and the two-factor 
interactions. The standard deviations for either the ME 
or 2FI are 3.0, 3.2, 3.0, 6.5, and 7.4 milli-pH units for 
solutions 5, 3, 2, 1, and 4, respectively. From this we can 
conclude that an effect or interaction of more than 20 
milli-pH units is not a chance happening. 

Looking at table 13, we see that main effects A (low- 
ering temperature), E (adding NaN0 3 ), and F (adding 
KC1) are large. The NaN0 3 main effect is about 30 for 
HC1 solutions 5, 3, and 4, and is about 55 for H 2 S0 4 
solutions 2 and 1. The KC1 ME is about 85 for HC1 
solutions 5 and 3, and 115 for the H 2 S0 4 solutions. We 
note that the glass electrode #2 gives a KC1 ME of only 
18 for HC1 solution 4. This result is not a fluke, but is the 
combined result from four separate experiments. 

Two-factor interactions of appreciable size are ob- 
served under the column headings C, D, and G. Table 5 
lists the possible 2FI interactions. It may be reasonable 
to assume that the 2FI associated with column C is due 
to the AE and/or BF interactions (temperature with 
NaN0 3 and/or stirring with KC1). It is known from ex- 
perience that stirring can influence pH measurement 



Table 11. Properties of acid solutions tested. 









Ionic 


Electrodes used (#1 or #2) 


Label 


pH 


Acid 


Contaminants 


and 1984 Measurement Dates 


Soln. 5* 


3.0 


HC1 


Hi purity water 


#1 July 16-20 


#2 Aug. 10 


Soln. 3 


3.7 


'* 


11 »> »» 


#1 July 24-30 




Soln. 4 


4.4 


" 


i» »» »» 


- 


#2 July3l-Aug. 2 


Soln. 6 


5.0 


M 


»> »» )> 


#1 July 19-20 




Soln. 2 


3.6 


H2SO4 


20 ppm by wt in water 


#1 July 12-13 


#2 Aug. 21 


Soln. 1 


4.3 


H2SO4 


3 ppm by wt in water 


#1 July 9-12 


#2 Aug. 20 



* This solution was used for the above examples. 
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Table 12. Summary of pre-averages for main designs and reverse-sign designs. 

















Milli-pH 
C 


Units- 










Electrode 


Soln. 


Sets 


pH 


s 


A 


B 


D 


E 


F 


G 


Legend for + Factors 










Main Designs 
















5 


1 + 3 


3.0 


7 


45 


-4 


9 


21 


26 


81 


1 


A = 25 °C 




3 


1+3 


3.7 


8 


35 


-14 





30 


44 


94 


-15 


B = solution stirred 




6 


1+3 


5.0 


107 


-34 


-44 


-28 


30 


194 


158 


-98 


C = 0.5 mL dilution 




2 


1 + 3 


3.6 


7 


35 


-15 


1 


44 


60 


120 


-10 


D = 1 cm electrode immersion 




1 


1 + 3 


4.3 


15 


10 


-21 


-6 


39 


70 


116 


-18 


E = NaN0 3 added 
F = KC1 added 


2 


4 


1+3 


4.4 


14 


36 


-32 


-4 


-28 


28 


11 


-32 


G = pH measured at 10 minutes 



Reverse-Sign Designs 



1 5 


2 


3.0 


- 


62 


-3 


2 


-16 


26 


80 





1 3 


2+4 


3.7 


10 


40 


-16 


26 


-22 


28 


86 


6 


1 6 


2 


5.0 


- 


-35 


-123 


192 


-32 


70 


45 


134 


I 2 


2 


3.6 


— 


42 


-9 


23 


-44 


54 


130 


24 


1 1 


2 


4.3 


- 


17 


-21 


41 


-31 


45 


90 


28 


2 5 


4 


3.0 





55 


-6 


11 


-5 


4 


37 





2 4 


2+4 


4.4 


27 


29 


-14 


30 


-2 


19 


26 


8 


2 2 


4 


3.6 


— 


37 


-7 


12 


-7 


11 


33 


2 


2 1 


4 


4.3 


— 


27 


-9 


20 


1 


23 


30 






Electrode Soln. 



Table 13. Summary of results— main effects and two-factor interactions. 



pH Acid 



Milli-pH Units 

B C D E 



Legend for + Factors 













Main Effects 














1 


5 


3.0 


HC1 


7 


53 


-4 


5 


2 


26 


81 


1 


A = 25 °C 


1 


3 


3.7 


" 


9 


38 


-15 


13 


4 


36 


90 


-4 


B = solution stirred 


1 


2 


3.6 


H 2 S0 4 


7 


38 


-12 


12 





57 


125 


7 


C = 0.5 mL dilution 


1 


1 


4.3 




15 


14 


-21 


IS 


4 


58 


103 


5 


D = 1 cm electrode immersion 
E = NaN0 3 added 


2 


4 


4.4 


HC1 


21 


32 


-23 


13 


-15 


24 


IS 


-12 


F = KC1 added 

G — pH measured at 10 minutes 












Two-Factor Interactions 










1 


5 


3.0 


HC1 


7 


9 


1 


-3 


-18 





-1 


-1 




1 


3 


3.7 


" 


9 


2 


-1 


13 


-26 


-8 


-4 


10 




1 


2 


3.6 


H 2 S0 4 


7 


4 


3 


11 


-44 


-3 


5 


17 




1 


1 


4.3 


" 


15 


4 





24 


-35 


-12 


-13 


23 




2 


4 


4.4 


HC1 


21 


-4 


9 


17 


13 


-4 


8 


20 
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results. The temperature, NaN0 3 , and KC1 are large ef- 
fects, and it seems reasonable to assume that these effects 
may also be involved in the larger 2FI. By similar rea- 
soning, the 2FI associated with column G may be due to 
AF and/or BE interactions (temperature with KC1 
and/or stirring with NaN0 3 ). The D column 2FI may be 
due to AB and/or EF interactions (temperature with 
stirring and/or NaN0 3 with KC1). It is interesting to 
note that the column D 2FI are appreciably different 
between the glass electrodes (#1 and #2), and also that 
the KC1 ME with the HC1 solutions is different between 
the two electrodes. 

We were surprised by the slightly different character- 
istics exhibited by the two glass electrodes. In general, 
each laboratory will have its own set of surprises. The 
point to be made is that in ruggedness testing one should 
widely explore for factors that can influence the mea- 
surement results. Conversely, it seems wise not to be too 
concerned with determining higher order interactions, 
or with making extremely precise determinations of the 
main effects and two-factor interactions. 

Can a factor such as the two different glass electrodes 
be used as a factor in the PB-design? It is certainly not 
a quantitative factor, such as temperature, and we can- 
not set the electrodes at a high or a low level. Never- 
theless, we can accept the two levels we happen to get, 
and can go through all of the calculations. The advan- 
tages of orthogonal effects are maintained. With 100% 
hindsight, we see that it would have been wise to in- 
clude the two different glass electrodes as one of the 
factors in the PB-design. As it turned out, we stumbled 
onto this factor by making a sequential series of PB- 
experiments. 

How should we react to the observation of an im- 
portant nonquantitative factor such as the glass elec- 



trode? We cannot easily tighten the glass electrode spec- 
ifications. Clearly, further exploratory work dealing 
with the basic science of the glass electrode mea- 
surement process is needed. 

As a final note we present table 14 which suggests an 
upper limit for precise pH measurements of a fully dis- 
sociated acid such as HC1. The results shown in table 
14, which were collected from previously listed tables, 
suggest an upper pH limit of about 4.5. Higher pH mea- 
surements, associated with solutions 6, are plagued by 
the solution's absorption of CO2 from the atmosphere. 
Special measurement procedures are required for pH 
measurements above 4.5. 

Table 14. Effect of pH on standard deviation. 



pH 


s (milli-pH units) 


3.0 


7 


3.6 


7 


3.7 


9 


4.3 


15 


4.4 


22 


5.0 


107 




Conclusions 



The principles of ruggedness testing involving the 
recognition of interactions have been discussed and il- 
lustrated by a study of pH measurements using glass 
electrodes. Separation of the main effects and the two- 
factor interactions was obtained. The statistical signifi- 
cance of individual main effects, and of groups of two- 
factor interactions was determined. The precision of the 
pH measurements was found to be a strong function of 
the pH level. The physical interpretation of these find- 
ings is further discussed in companion articles, [5] in 
particular, of the NBS Journal of Research. 
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